How should we run the EA Forum Prize?
After collecting feedback from the community, I’ve decided to make June the last month that CEA will definitely award the EA Forum Prize in its current form. This means that we will publish prize posts for May 2020 and June 2020, but won’t necessarily award prizes after that. (This was also announced in the most recent prize post.)
Overall, while some users reported finding the Prize valuable or motivating, that number wasn’t quite as high as I had been hoping for. I also heard some good suggestions for ways the Prize could be restructured.
This thread is meant to collect additional suggestions for how we could run the Forum Prize — or otherwise use the funding (roughly $20,000/year) to promote the creation of excellent Forum content.
My question: How would you suggest we run the Prize, or otherwise use the aforementioned budget to support content creation?
(Note that we may wind up sticking to the same format, or something very similar, if we don’t find a different arrangement that we like more.)
If you have thoughts on any of the ideas below, or other suggestions of your own, please let us know on this post! You can also contact me if you’d prefer privacy.
Ideas we’ve heard or thought of so far
Themed prizes (e.g. setting aside one prize for “the best post on topic X”)
Giving more prizes, even if they end up being smaller on average
Selecting at least one judge to represent each major EA cause area
Including a community vote (not just upvotes, but a separate voting process). This would likely supplement judges’ votes, not replace them.
Having a special “first post” and/or “first comment” prize for people who make a really good first contribution from an account
Having separate prizes for orgs/professional researchers and people who contribute to the Forum on more of an “amateur” basis
Having more flexible prize amounts (e.g. maybe one post should win all the money in some months if it’s especially good, or maybe money should be distributed according to vote ratios rather than just first/second/third place)
Having judges who are somewhat removed the community (or finding some other way to reduce the extent to which the Prize may reflect the biases of the community or of central orgs within the community)
Getting rid of the Prize entirely without replacing it (one survey respondent, who has written many excellent posts and won at least one Prize, believes it to be “distracting and unnecessarily divisive”)
- EA Forum Prize: Winners for May 2020 by (21 Jul 2020 7:53 UTC; 38 points)
- 's comment on NunoSempere’s Quick takes by (8 Nov 2020 11:42 UTC; 18 points)
- 's comment on Does the Forum Prize lead people to write more posts? by (21 Sep 2021 12:38 UTC; 4 points)
- 's comment on Rowing and Steering the Effective Altruism Movement by (14 Jan 2022 9:57 UTC; 4 points)
Regarding professional vs amatur researchers, I think that a technically easier distinction would be of posts that were written as a part of some organisation’s work or independently. (So that professional researchers in EA Orgs would still compete for the prize like anyone else). This makes sense, as they got paid for that work and probably would have done it anyway regardless of the forum prize. This would apply to people working on funded independent projects too.
I think that other methods mentioned in the post would be better suited for leveling the playing field.
Personally I find the prize disproportionately motivating, in that it increases my desire to write EA forum content to a level beyond what I think I’d endorse if I reflected for longer.
Sorry if this is not very helpful; I imagine it’s also not very representative.
General comments
In general, I am in favour of having a forum prize of some kind. I personally find it motivating and I think it is a useful way for people to gain credibility within the EA movement. I think it also helps lend the forum an air of seriousness, which is sometimes good and sometimes bad but does probably raise the quality of the content.
From the “giving individuals credibility” perspective, having a few large prizes is good (a $750 prize is large enough to put on a CV for many people, a $50 one is clearly not). From the “motivating good content” perspective, having a moderately larger number of moderately smaller prizes would probably be better.
Perhaps I’m wrong about this, but I’ve sometimes felt like comment prizes are used to make up the difference on post prizes (“your post was very good but not top-3 this month, so we found a decent comment you made on that post and gave it to that!”). If so I think that is good evidence that we want more smaller prizes.
Comments on specific proposals
This seems like something you might want to do less frequently (annually?) in addition to more regular general prizes.
On the whole I think this would be a good idea (see above).
I don’t feel like any major cause area has been getting shafted here; I can readily recall winners on global health, animal welfare, long-term future, and meta topics. In the absence of a specific problem this seems like not a good idea; I feel like it would not provide much benefit and might encourage tribalism/factionalism.
The above comments assume topic-general prizes like the current system; I think themed prizes probably would want some specialist judges involved.
A lot of award ceremonies have some kind of “people’s choice” award. If you do this I again think it should probably be less frequent than monthly; perhaps quarterly or annually.
Meh. This sounds cute but I don’t actually think it would be very valuable, and could actually be harmful on net. I think it’s good to set a high quality bar and incentivise people to work up to it, rather than agonising over making their first post perfect even more than they probably already do.
So I suppose the argument for this is that some people basically get paid to write on the Forum, and it’s not surprising that those people (coughSauliuscough) win lots of prizes, which crowds out good content from amateurs. I’m not sure to what extent this is true, and if true, I’m unconvinced that it’s bad.
Firstly, a lot of that content is just very good and I want there to be strong incentives to get that stuff on the forum. Secondly, a lot of prize-winning posts are from people who work for EA Orgs but are writing in their private capacity; indeed, you’d expect people who are especially committed to thinking about EA topics to both be more likely to win Forum prizes and be more likely to work for an EA Org. This seems like a good thing we want more of, no less so than contributions from non-Org-employees, so I don’t think they should all be put in a special category and forced to compete with each other.
There’s a similar argument which is something like “separate out content people write for work vs for pleasure”. I think this is more defensible but still probably wrong. If prizes incentivise orgs to post more of their work on the EA Forum, this seems like a good thing that benefits everybody. But I wouldn’t be extremely surprised to be argued out of this.
This sounds bad to me, but for fairly vague reasons. I feel like it gives the judges too many degrees of freedom, and that it’s probably good practice to have prize amounts be fairly predictable. But I’m not sure about this.
This sounds like a great way to remove most of the value of the prize in return for highly dubious gains.
If I look at the winners of the EA Forum Prize, I expect to see exemplars of great EA content. The best people to judge great EA content are EAs, i.e. community members. We could bring in external expects from adjacent fields (global health, AI, etc), but why?
Firstly, those people are likely to be far more limited in the range of posts they can judge, which basically forces us into the “subject-specific prizes” model above; while I think those could be good to have occasionally I wouldn’t want them to completely take over the prize.
Secondly, those people are almost certain to be less well-aligned with the distinctive values of the EA community, which makes their opinions on which posts should get prizes much less valuable in this particular context.
Thirdly, if we do want domain experts to judge a domain-specific EA prize I’m pretty sure we can find them within the EA community, rather than bending over backwards to make them external.
And finally, all those fields already have their own fora, their own prizes, and their own way of sharing and evaluating information. It’s fine – indeed, very good – for the particular nexus known as EA to have their own systems and prizes too. It only becomes a problem if we think EA Forum prizes are the sole arbiters of truth and quality.
Has there been any evidence of the prize being divisive, i.e. actually causing conflict? Perhaps there is, but I’m not aware of it. And calling it distracting is confusing to me; is the claim that it incentivises people to write content based on a Keynesian beauty contest, rather than what they actually think it would be best for them to write?
Anyway, I don’t have much sympathy with that claim as stated, or for abolishing the prize entirely on that basis, but there might be an alternative interpretation that I’d be more sympathetic to.
I went and checked and I think the people who have won the most money from the EA Forum Prize are actually Buck Shlegeris and Ben Kuhn (tied for first place). I don’t know if they were paid to write what they wrote but my guess is they were not.
Why don’t you conduct an experiment? E.g. you could award prizes only for posts/comments written by users whose usernames start with letters A-L (and whose accounts were created prior to the announcement) and see if you notice any significant difference in the quality of those users’ submissions.
I’ll offer a data point: I’m not particularly motivated to post on the Forum by a monetary prize. It hasn’t led me to post on the Forum more than I ordinarily would. I am somewhat interested in social rewards, but the karma feature seems to do that better than the Forum Prize.
Also, as someone who doesn’t read every single post on the Forum, I also find the Prize useful for highlighting what content is actually worth reading, but again, I think highlighting posts based on karma instead (with or without a monetary prize) would work just as well.
If the Forum Prize does continue, I do think there should be separate categories for professional researchers and “amateurs.”
I totally agree with this:
On the other hand, I think high-karma posts are already in evidence in the Forum Favorites section.
So for me the prize fulfills some very important purposes. Perhaps the most important two are:
Curating the best content
Rewarding content creators for producing content
----------------------------------
Curating the best content
I regularly use the prize posts as a “summary of the best of the month” which I greatly appreciate. It helps me focus my attention on the best articles of the month. It is also a great experience for the authors, who just publish content as usual and without any additional overhead sometimes they get selected for the prize. This is how I wish more academic areas worked—everything published openly in a preprint archive and then the journals acting as “curators”, selecting the best work. I really wish something like this will remain in the forum (a “best of the month” selection).
This is a very useful function also to help analyze a posteriori the impact of the best pieces of the forum, as for example with this post.
-----------------------------------------------
Rewarding our content creators for producing content
I would think some people who are specially competitive are motivated by the prize to write more. But I don’t know how large of a share of the community is like that.
Instead, I think the most important reward is making people feel proud and recognized for their work. When somebody I knew won the forum or comment prize, they were showered with praise and felt happy and appreciated.
This goal is a bit at tension with the goal of curating the best content. People are sometimes disheartened by the posts being consistently won by professional researchers who have the time and experience to write very good posts.
I think I would like this goals to be somehow separated—though I admit I am a bit confused about how one would go about doing that.
One specific point about the “first post” / “first comment” idea:
If someone asked me for advice on how to get good at writing EA Forum posts, the most succinct advice I could give is this:
write!
You get better by doing something by practicing, so I would encourage someone to produce more content.
Which seems to suggest that a first post/comment prize is bad because it holds people back from taking the first step, and therefore discourages people from getting better.
I appreciate that there’s a counterargument to this (we don’t want to flood the forum with bad content), but I don’t feel like we have that a problem with this now, so I’m less worried about the counter.
I agree with this and also expect a “best first post” comment to probably be net harmful.
If there were to be some sort of “best new user” prize, I think it should probably be awarded less frequently, say every 6 or 12 months, for someone whose first post was within that window. I still think this would probably not be the best use of the available funding, but it seems to fit the spirit of the “best first post” idea while avoiding some of the worst side-effects.
In my case at least, I can say that both the money and the reputation for winning was extremely valuable. Thanks for that!
Maybe more but smaller prizes? I think karma works as a better signal to internalize since the feedback is more consistent and granular, but if more prizes were given out, this gap would be smaller. Maybe reduce the post prizes to $150 to $300 and give out more of them and/or more comment prizes?
If someone is working on a post, I don’t think $200 vs $500 makes much difference in motivation beyond the karma on the post and recognition with the prize, but 0 vs $200 could (although I don’t know that either would make much difference for me beyond the karma).
As discussed in a bit more detail in this post, I’d love to see themed prizes focusing specifically on critical engagement with effective altruism. This could be very broad (e.g., “Best critique of the effective altruism movement”) or more narrow (e.g., something like “Best critique of a specific assumption that is widely made in the community” or “Best writeup on how applied longtermism could go wrong”).
To the next content specialist on the Forum: I’d be happy to discuss further!
Almost all content useful to EAs is not written on the forum, and almost all authors who could write such content will not write it on the forum. So it would be a lot more valuable to reward good content whether or not it is on the forum. It is harder to evaluate all content, but one can consider nominated content. If this is outside one’s job description, then can one change the job description?
Could you say more about the other sources you are thinking of as better locations? Are they aggregators or are you just thinking about the long tail of e.g. blogs hosted by individual organizations or academic journals?
Crossposting to the Forum is one obvious way that externally hosted content could get here, though it’s inelegant.
Yeah, I think high-quality content is spread across many blogs. But not terribly hard to find—a lot of it is in blog posts that can be seen by following a hundred Twitter accounts.
I agree crossposting or linkposting is one way to gather content. I guess that’s kind-of what subreddits/hackernews/Twitter all do, but those platforms are more-designed for that purpose. Not sure what’s the best solution.
What are some examples of blog content you think is at least as good as a very good Forum post, is potentially useful to EAs, and wasn’t crossposted to the Forum?
Ideally, if I find content like that, I’d like to crosspost it for people to read here, rather than hoping that Forum readers also start reading a bunch of different blogs as a result of seeing them in Prize writeups. I think your initial comment is a good argument for more crossposting, or perhaps having a “suggest crosspost” box people can use to send me URLs of things they’d like posted here (if they want to save time on formatting and such).
I don’t think rewarding content outside the Forum is necessarily “a lot more valuable” unless doing so gets those authors to write more EA-friendly content and/or use the Forum more. To the extent that the Prize is an incentive rather than just a reward, it seems like a better incentive to offer more predictable rewards to people who submit their writing to a single website rather than rewarding people from all over the web.
To draw a comparison: If a literary journal pays people who submit great stories to the journal, is that going to incentivize more good short-fiction writing than going to authors who don’t know about the journal and giving them money for things they wrote elsewhere? I don’t think the answer is obvious, and I’d lean toward the former being more useful.
In the last month or so, here are a bunch of things I’ve enjoyed reading that weren’t on the forum:
Blogs:
http://www.paulgraham.com/orth.html
https://scholars-stage.blogspot.com/2020/07/why-banning-all-party-members-is-stupid.html
https://scholars-stage.blogspot.com/2020/07/the-world-that-twitter-made.html
https://stefanfschubert.com/blog/2020/7/24/the-focus-of-collective-attention-and-the-long-run-future
https://www.lesswrong.com/posts/N6vZEnCn6A95Xn39p/are-we-in-an-ai-overhang
News (opinion):
https://www.bloomberg.com/opinion/articles/2020-07-29/china-s-xi-jinping-could-make-same-mistakes-as-kaiser-wilhelm-ii?fbclid=IwAR3xWbe7IcvfVul4tqv6a1psRU8gLAjnyamL0wysI8fpBqIT3OPvoKRPO7A
Other:
https://www.gwern.net/GPT-3
https://ourworldindata.org/coronavirus
http://www.usrsb.in/three-gorges-dam-2020.html
https://theprecipice.com/quotations
Here is the kind of initiative that would seem more useful to me than an EA forum prize (if you switched in EA for liberalism): https://mobile.twitter.com/tylercowen/status/1296152595728412675
This does seem like a very good initiative, though I wonder whether grant funding might make more sense than small prizes. Jason Crawford got $75,000 from Open Phil (and also an EA Funds grant), though I don’t know how many bloggers may have applied to EA Funds/been looked at by Open Phil and then turned down.
With Prize funding, we could support roughly $22,000 in grant funds per year, which could provide solid incentives to… a dozen bloggers, maybe? Something to think on.
Are there any blogs/newsletters/etc. that you’d fund if you had the money to run a Tyler Cowen-like initiative for EA writing?
Yeah I don’t know whether/when grants or prizes are better, or exactly what the optimal initial scale is, although presumably you would want to go beyond $22k/yr once it has been demonstrated to work. One would also want to look at why previous prizes, such as Paul’s alignment prizes didn’t work out.
I guess I would be granting to individuals similar to those who “I’ve enjoyed reading” according to the post above. I also wonder if someone could get Zach Weinersmith to do something EA, given how much related stuff he’s already done previously.
I think that this is not very relevant. If you have something to write which would be valuable to the EA community, when would you expect it to be highly valuable to publish it to the forum? How valuable do you think that is?
I am fairly confused by the strong (!) downvote on this comment.
Huh.. I think I may have been too rude, sorry if that’s the case.
To clarify, I think that the question of where the useful knowledge is, is a bit distracting from the question of whether good content should be written on the forum and also content which is directed at the EA community I’d expect to be best distributed through the forum itself.
I agree that more impact is probably done in professional settings where the topic is something related to solving one of the most important problems, if that content is strong enough, but I think that topics related to the EA community or to cause prioritization could have a lot of impact here. There is also good reasons to believe that writing cause-specific content here has impact in terms of collaborations and education. Also, I think that the forum is a great place to flesh out content which is not quite at the level of perfection needed for publications to get peer feedback.
The comment was probably strong-downvoted because it is confidently wrong in two dimensions:
1. The EA Forum only exists to promote impactful ideas. So to say that the question “where are impactful ideas?” is a distraction from the question “when should we post on the Forum?” is to have things entirely backwards. To promote good ideas, we do need to know where they are.
2. We are trying to address what a community-builder should do, not a content-creator. It is a non-sequitur to try to replace the important meta-questions of what infrastructure and incentives there should be, with the question of when an individual should post to the forum.
Ah, thanks! I think I was perceived as saying “content creators publish stuff which is highly valuable to the forum as well, and therefore we should give them a prize!”.
It is definitely not what I intended, and it was a very sloppy writing from me 🤦♂️
What I do think is that the forum is a good platform and that it makes total sense to optimise on building better incentives there. Not as a matter of job descriptions, but generally when building organisations and platforms I think that it makes sense to focus some of the efforts and resources on itself even if there might be alternatives which might be better for that org/platform’s stated goals but they do not really work together well with other parts of the org/platform.
The questions were there because I specifically didn’t understand why you think that “Almost all content useful to EAs is not written on the forum, and almost all authors who could write such content will not write it on the forum”, regardless of what we think about the previous point. I’m very interested in your take on that!
Me too. Maybe a normal downvote by a very high karma member? And I also remember one instance where someone accidentally clicked on downvote without noticing.
No user on the Forum has a “normal” vote worth more than 2 karma.
(The full karma system is written out in this post.)
Possibly, prizes can be directed to specific proposals. I can imagine a platform (which could be the comment section of a post) where people write what they intend to write about, or what they’d want someone else to write about, and people can comment and upvote on what seems to them important (and why). Out of these, several such proposals can be chosen and offered a bounty on them—which would be the maximal bounty if a thorough and well-written post would be written on them. Then, quality and how much the original question was solved would be assessed and the prize given would be the bounty times the precentage of the question solved times some quality factor.
I really like this idea. CEA could offer people they trust to use the Forum Prize money to announce prizes for exploring a topic or specific question and writing a post about it.
Why this might be a good idea
1. incentivize useful content (and others could chime in and add to the prize pool, signalling higher usefulness)
2. direct attention for people looking for useful work they might do as a sideproject
Some possible downsides
1. Draws in people that are not primarily altruistically motivated
could be an upside, too, if they do great work and get drawn in, as I expect most smart people will be happy to be able to contribute to improve the world
2. Disagreements about fulfilling the goal criteria might cause frustration
3. Increase the noise in the forum, i.e. more posts of lower quality
Skimming a bit through the forum, this idea has been tried a few times! But less often then I’d have expected if this were a good idea. Some examples:
Call for: Essay on “How to evaluate small-scale giving opportunities”, prize: 300$, result: no entries in time (two commenters signalling interest)
Call for: Arguments/comments for or against donating to the EA Hotel, prize: 100$, result: 35 comments, looks good
Call for: Arguments/comments if psychedelics are a potential EA cause area, prize: 400$, 200$, 100$, result: 114 comments, looks good, too
Call for: “Spaced Repetition literature review” on LW, prize: 385$, result: 2 reviews, Gwern won (and the prize seemingly motivating him to write it)
Call for: Essay Contest: Cryonics and Effective Altruism, prize: 1BTC (then 350$), result: 5 essays, all rated positively
I just found out that there is a tag on LW for active and closed bounties, listing 4 and 24 prizes in total. My impression is that experimenting with bounties on the EA forum would be a good idea and I wonder what others here think about it.
This can cause perverse incentives. People can hold back on a written post until a bounty is made for that post. People would disproportionally compete on working on the selected issues. People might be discouraged from starting a post they know they won’t get a prize for
A different approach. It seems that the prize is designed to improve incentives for writing high-quality posts. This is arguably not as important as other routes for better information flow, communication, coordination and epistemics. Perhaps people are already motivated enough to write posts, and are motivated enough to write them well. Perhaps they are even motivated for either of these too strongly (as opposed to other means of impacts and fleshing out less baked thoughts).
So several alternatives might be:
prize for best distillation, explanation or collection
prize for best (unfunded) volunteer community project
prize for best public conversation/debate/anti-debate.
prize for most important errors in arguments.
prize for general supportive and welcoming comments by a user over different occasions.